Project Description¶
Background & Context¶
The Thera bank recently saw a steep decline in the number of users of their credit card, credit cards are a good source of income for banks because of different kinds of fees charged by the banks like annual fees, balance transfer fees, and cash advance fees, late payment fees, foreign transaction fees, and others. Some fees are charged on every user irrespective of usage, while others are charged under specified circumstances.
Customers’ leaving credit cards services would lead bank to loss, so the bank wants to analyze the data of customers’ and identify the customers who will leave their credit card services and reason for same – so that bank could improve upon those areas
You as a Data scientist at Thera bank need to come up with a classification model that will help bank improve their services so that customers do not renounce their credit cards
Objective¶
- Explore and visualize the dataset.
- Build a classification model to predict if the customer is going to churn or not
- Optimize the model using appropriate techniques
- Generate a set of insights and recommendations that will help the bank
Data Dictionary¶
- CLIENTNUM: Client number. Unique identifier for the customer holding the account
- Attrition_Flag: Internal event (customer activity) variable - if the account is closed then 1 else 0
- Customer_Age: Age in Years
- Gender: Gender of the account holder
- Dependent_count: Number of dependents
- Education_Level: Educational Qualification of the account holder
- Marital_Status: Marital Status of the account holder
- Income_Category: Annual Income Category of the account holder
- Card_Category: Type of Card
- Months_on_book: Period of relationship with the bank
- Total_Relationship_Count: Total no. of products held by the customer
- Months_Inactive_12_mon: No. of months inactive in the last 12 months
- Contacts_Count_12_mon: No. of Contacts in the last 12 months
- Credit_Limit: Credit Limit on the Credit Card
- Total_Revolving_Bal: Total Revolving Balance on the Credit Card
- Avg_Open_To_Buy: Open to Buy Credit Line (Average of last 12 months)
- Total_Amt_Chng_Q4_Q1: Change in Transaction Amount (Q4 over Q1)
- Total_Trans_Amt: Total Transaction Amount (Last 12 months)
- Total_Trans_Ct: Total Transaction Count (Last 12 months)
- Total_Ct_Chng_Q4_Q1: Change in Transaction Count (Q4 over Q1)
- Avg_Utilization_Ratio: Average Card Utilization Ratio
Importing libraries¶
In [1]:
import warnings
warnings.filterwarnings("ignore")
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
from sklearn import metrics
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.impute import KNNImputer
from sklearn.pipeline import Pipeline, make_pipeline
#libraries to help with model building
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import (
AdaBoostClassifier,
GradientBoostingClassifier,
RandomForestClassifier,
BaggingClassifier)
from xgboost import XGBClassifier
from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import RandomUnderSampler
from scipy.stats import randint as sp_randint
Loading dataset¶
In [2]:
dataCC = pd.read_csv('BankChurners.csv')
In [3]:
#taking a copy of the dataset in case we need original data later on
data = dataCC.copy()
View first 5 and last 5 rows of the dataset¶
In [4]:
dataCC.head()
Out[4]:
In [5]:
dataCC.tail()
Out[5]:
Checking shape of dataset¶
In [6]:
dataCC.shape
Out[6]:
Checking data types and number of non-null values for each column¶
In [7]:
dataCC.info()
Insights:¶
- Dataset has 10127 rows and 22 columns (including unnamed column which can be removed)
- Column 'Unnamed: 21' has all null values and can be dropped from the dataset
- None of the columns seem to have null/missing values except 'Unnamed: 21', which is an empty column
- The columns are integer, float or object types
- Different features seem to be on different scales
NULL (missing value) check¶
In [8]:
#Just checking null count, even though we saw there are no nulls except for 'Unnamed: 21'
dataCC.isnull().sum().sort_values(ascending=False)
Out[8]:
Removing unnamed column¶
In [9]:
#Since 'Unnamed: 21' is an empty column, it can be dropped
dataCC.drop('Unnamed: 21',axis=1,inplace=True)
In [10]:
#Checking null count and shape again to confirm empty column is dropped
dataCC.isnull().sum().sort_values(ascending=False)
Out[10]:
In [11]:
dataCC.shape
Out[11]:
- NULL values are not present in the dataset now
Number of unique values in each column¶
In [12]:
dataCC.nunique().sort_values(ascending=False)
Out[12]:
Insights:¶
- Client Number (CLIENTNUM) is an ID column and won't add value to the model, it can be dropped
- Avg_Open_To_Buy,Credit_Limit,Total_Trans_Amt,Total_Revolving_Bal,Total_Amt_Chng_Q4_Q1,Avg_Utilization_Ratio, Total_Ct_Chng_Q4_Q1,Total_Trans_Ct,Customer_Age,Months_on_book seem to have continuos values
- Contacts_Count_12_mon,Months_Inactive_12_mon,Education_Level,Income_Category,Dependent_count,Total_Relationship_Count, Card_Category,Marital_Status,Gender,Attrition_Flag seem to be categorical from the number of unique values
In [13]:
#Dropping column CLIENTNUM
dataCC.drop('CLIENTNUM',axis=1,inplace=True)
In [14]:
#Checking shape again to make sure that CLIENTNUM is dropped
dataCC.shape
Out[14]:
Summary of the dataset¶
In [15]:
#This gives the summary for numeric (int/float) columns
dataCC.describe().T
Out[15]:
- Customer_Age ranges from 26 to 73, mean and median are almost same (46), fairly normal distribution
- Dependent_count ranges from 0 to 5, mean and median are almost same (2), fairly normal distribution
- Months_on_book ranges from 13 to 56, not much difference in mean and median
- Total_Relationship_Count ranges from 1 to 6, mean and median are almost same (4)
- Months_Inactive_12_mon ranges from 0 to 6, mean and median are almost same (2)
- Contacts_Count_12_mon ranges from 0 to 6, not much difference in mean and median
- Credit_Limit ranges from 1438 to 34516 (a large range), mean way more than median, right skewed data, and there is a big difference between 75% and max values, which indicates possible outliers towards the right end
- Total_Revolving_Bal ranges from 0 to 2517 (a large range), not a huge difference b/w mean and median, there is some difference b/w 75% and max, more data points are there b/w 25% and 50%
- Avg_Open_To_Buy ranges from 3 to 34516 (really large range), mean way more than median, right skewed data,and there is a big difference between 75% and max values, which indicates possible outliers towards the right end
- Total_Amt_Chng_Q4_Q1 ranges from 0 to 3.397, mean and median almost same
- Total_Trans_Amt ranges from 510 to 18484 (a large range), not much difference in mean and median, there is some difference b/w 75% and max, possible outliers towards the right end
- Total_Trans_Ct ranges from 10 to 139, not a big difference b/w mean and median
- Total_Ct_Chng_Q4_Q1 ranges from 0 to 3.714, mean and median almost same
- Avg_Utilization_Ratio ranges from 0 to 0.999, mean > median
In [16]:
#This gives the summary for non-numeric (object) columns
dataCC.describe(include=['object']).T
Out[16]:
- All object columns look like category, from number of unique values
- No null values in these object type columns
- Attrition_Flag, which is our target variable has 2 unique values and record count is more for 'Existing Customer'
- Gender has 2 unique values and record count is more for 'Female'
- Education_Level has 7 unique values and record count is more for 'Graduate'
- Marital_Status has 4 unique values and record count is more for 'Married'
- Income_Category has 6 unique values and record count is more for 'Less than 40K'
- Card_Category has 4 unique values and record count is more for 'Blue'
- We will explore these columns further using univariate and bivariate analysis
Checking for duplicate records in the dataset¶
In [17]:
dataCC.duplicated().sum()
Out[17]:
- No duplicates found in the dataset
Checking the count of each unique category in categorical variables¶
In [18]:
cat_cols = ['Contacts_Count_12_mon','Months_Inactive_12_mon','Education_Level','Income_Category',
'Dependent_count','Total_Relationship_Count','Card_Category','Marital_Status','Gender','Attrition_Flag']
for col in cat_cols:
print(dataCC[col].value_counts())
print('-'*50)
- Unique values for 'Number of contacts in last 12 months' are 0,1,2,3,4,5,6. And for 3 contacts in last 12 months, we have more data points followed by 2
- Unique values for 'Number of months inactive in last 12 months' are 0,1,2,3,4,5,6. And we have more data points for 3 inactive months in last 12 months followed by 2
- Unique values for 'Education Level' are Graduate, High School, Unknown, Uneducated, College, Post-Graduate and Doctorate. Unknown seems to be missing value. Data points are more for Graduate followed by High School
- Unique values for 'Income Category' are Less than 40K, 40-60K, 60-80K, 80-120K, 120K+ and Unknown. Again unknown seems to be missing value. Data points are more for 'Less than 40K' followed by 40-60K
- Unique values for 'Number of dependents' are 0,1,2,3,4,5. Data points are more for 3 dependents followed by 2
- Unique values for 'Total number of products held by the customer' are 1,2,3,4,5,6. Data points are more for 3 products followed by 4
- 'Card Category' has 4 unique values - Blue, Silver, Gold, Platinum. Blue has more data points, followed by silver
- Married, Single, Divorced, Unknown are unique values in 'Marital Status' feature. 'unknown' looks like missing value. Data points are more for Married followed by Single
- Female customer data is more compared to Male
- Attrition_Flag which is our target variable seems to be NOT balanced. Existing Customer data (+ve class) is more compared to Attrited Customers (-ve class). But -ve class is of more focus in this project, since it tries to predict if the customer is going to churn or not
In [19]:
#Converting object types to category
for col in dataCC.columns:
if dataCC[col].dtype == 'object':
dataCC[col] = dataCC[col].astype('category')
- Based on number of unique values, following features have only handful of unique values and can be thought of as categories
But these features can get more values if the dataset grows and hence going to treat them as numerical when building models
- Dependent_count
- Total_Relationship_Count
- Months_Inactive_12_mon
- Contacts_Count_12_mon
In [20]:
#checking info() again to see the data types
dataCC.info()
- Reduces memory usuage by converting to category
EDA - univariate analysis¶
Univariate analysis for continuous variables using boxplot and histogram¶
In [21]:
# Function to create boxplot and histogram for any input numerical variable
# This function takes the numerical column as the input and returns the boxplots
# and histograms for the variable.
def histogram_boxplot(feature, figsize=(15,10), bins = None):
""" Boxplot and histogram combined
feature: 1-d feature array
figsize: size of fig (default (9,8))
bins: number of bins (default None / auto)
"""
f2, (ax_box2, ax_hist2) = plt.subplots(nrows = 2, # Number of rows of the subplot grid= 2
sharex = True, # x-axis will be shared among all subplots
gridspec_kw = {"height_ratios": (.25, .75)},
figsize = figsize
) # creating the 2 subplots
sns.boxplot(feature, ax=ax_box2, showmeans=True, color='violet') # boxplot will be created and a star will indicate the mean value of the column
sns.distplot(feature, kde=F, ax=ax_hist2, bins=bins,palette="winter") if bins else sns.distplot(feature, kde=False, ax=ax_hist2) # For histogram
ax_hist2.axvline(np.mean(feature), color='green', linestyle='--') # Add mean to the histogram
ax_hist2.axvline(np.median(feature), color='black', linestyle='-') # Add median to the histogram
In [22]:
#Customer_Age
histogram_boxplot(dataCC['Customer_Age'])
- Mean and Median are very close to each other
- Fairly normal distribution of data is seen
- There are couple of outliers towards the right end
- We are going to see the outliers next
In [23]:
dataCC[dataCC['Customer_Age'] > 65 ]['Customer_Age'].sort_values(ascending=False)
Out[23]:
- Does not look like the two points that we see in boxplot as outliers need to be treated as outliers, since there is not a big difference between the data point before those two points and these two right most points.
In [24]:
#Months_on_book
histogram_boxplot(dataCC['Months_on_book'])
- Mean and Median are almost the same
- Except for the peak between 30 and 40, it shows a fairly uniform distribution
- Boxplot shows couple of outliers on both the left and right end
- Let us see if these are outliers that need to be considered as outliers
In [25]:
dataCC[dataCC['Months_on_book'] >= 50 ]['Months_on_book'].sort_values(ascending=False)
Out[25]:
In [26]:
dataCC[dataCC['Months_on_book'] < 20 ]['Months_on_book'].sort_values(ascending=True)
Out[26]:
- Does not look like the points that we see in boxplot as outliers need to be treated as outliers, since there is not a big difference between the data point before/after those points and these outlier points.
In [27]:
#Credit_Limit
histogram_boxplot(dataCC['Credit_Limit'])
- Mean way more than median and the data is right skewed
- From the boxplot, we can see that there are lot of outliers towards the right end
- There is a big difference between 75% and the max data points too
- We are going to see the outliers separely
In [28]:
dataCC[dataCC['Credit_Limit'] > 25000 ]['Credit_Limit'].sort_values(ascending=False)
Out[28]:
- Some variation is always expected in real world scenarios for the income variable and here the right most end data points are many, they are not completely standing out from the rest of the data.
- So it does not look like data entry errors and need not be removed
In [29]:
#Total_Revolving_Bal
histogram_boxplot(dataCC['Total_Revolving_Bal'])
- Slightly left skewed, except for the peak at the lowest end
- Mean and Median are not much different
- Most of the data is concentrated b/w 25% and median
- Boxplot does not show any outliers
In [30]:
#Avg_Open_To_Buy
histogram_boxplot(dataCC['Avg_Open_To_Buy'])
- Right skewed data distribution
- Boxplot shows outliers towards the right end of the distribution
- Most data points are towards the lower end of the distribution, which is kind of expected in real world scenarios as the spending will be more
- Let us check the outliers to see if there are any unusual data points
In [31]:
dataCC[dataCC['Credit_Limit'] >= 30000 ]['Credit_Limit'].sort_values(ascending=False)
Out[31]:
- There are many data points for the value '34516' itself, which is the max value
- Since these are not standing out from the rest of the data, this does not look like data entry errors and need not be removed
In [32]:
#Total_Amt_Chng_Q4_Q1
histogram_boxplot(dataCC['Total_Amt_Chng_Q4_Q1'])
- Mean and Median almost same
- The distribution is fairly normal, if we don't consider the outliers
- Many outliers both at the left and right ends
- Let us check the outliers
In [33]:
dataCC[dataCC['Total_Amt_Chng_Q4_Q1'] > 2.0 ]['Total_Amt_Chng_Q4_Q1'].sort_values(ascending=False)
Out[33]:
In [34]:
dataCC[dataCC['Total_Amt_Chng_Q4_Q1'] < 0.2 ]['Total_Amt_Chng_Q4_Q1'].sort_values(ascending=True)
Out[34]:
- There are 2 data points greater than 3.0 and are part of the outliers at the right end
- There are 5 data points that are 0.0 and are part of the outliers at the left end
- These are not standing out from rest of the values and hence need not be removed
- There can be variations in change in transaction amount in real world scenarios too
In [35]:
#Total_Trans_Amt
histogram_boxplot(dataCC['Total_Trans_Amt'])
- Right skewed data distribution, with lot of outliers towards the right end
- There is significant difference b/w 75% and max data points, most of the data points concentrated at the lower end
- It is good to notice that there are no data points with 0 transaction amount
In [36]:
dataCC[(dataCC['Total_Trans_Amt'] > 7500) & (dataCC['Total_Trans_Amt'] < 10000) ]['Total_Trans_Amt'].sort_values(ascending=False)
Out[36]:
In [37]:
dataCC[(dataCC['Total_Trans_Amt'] > 12500) & (dataCC['Total_Trans_Amt'] < 17000) ]['Total_Trans_Amt'].sort_values(ascending=False)
Out[37]:
- There are lot of data points in these outliers and does not seem to be data entry errors
- And in real world scenario too total transactiom amount can vary
- Hence deciding to keep the outliers
In [38]:
#Total_Trans_Ct
histogram_boxplot(dataCC['Total_Trans_Ct'])
- Mean and Median are almost close to each other
- Except for some peaks, the distribution is fairly normal
- Boxplot shows couple of outliers at the right end
In [39]:
dataCC[dataCC['Total_Trans_Ct'] > 135 ]['Total_Trans_Ct'].sort_values(ascending=False)
Out[39]:
- These are not really standing out from the rest of the data
- Deciding to keep the points and not removing them
In [40]:
#Total_Ct_Chng_Q4_Q1
histogram_boxplot(dataCC['Total_Ct_Chng_Q4_Q1'])
- Mean and Median almost same
- The distribution is fairly normal, if we don't consider the outliers
- Many outliers both at the left and right ends
- Let us check the outliers
In [41]:
dataCC[dataCC['Total_Ct_Chng_Q4_Q1'] > 2.0 ]['Total_Ct_Chng_Q4_Q1'].sort_values(ascending=False)
Out[41]:
In [42]:
dataCC[dataCC['Total_Ct_Chng_Q4_Q1'] < 0.2 ]['Total_Ct_Chng_Q4_Q1'].sort_values(ascending=True)
Out[42]:
- There are many data points at the left and right ends
- Since these are not standing out from the rest of the data, this does not look like data entry errors and need not be removed
In [43]:
#Avg_Utilization_Ratio
histogram_boxplot(dataCC['Avg_Utilization_Ratio'])
- Mean is greater than Median and it is right skewed data
- There are no outliers
- More data points are below 0.2, which means the avg utilization is really low
Univariate analysis for categorical variables using bar charts¶
In [44]:
#Function to plot barchart with percentage for categorical variables
def perc_on_bar(feature):
'''
plot
feature: categorical feature
the function won't work if a column is passed in hue parameter
'''
#Creating a countplot for the feature
sns.set(rc={'figure.figsize':(15,7)})
ax=sns.countplot(x=feature, data=dataCC)
total = len(feature) # length of the column
for p in ax.patches:
percentage = '{:.1f}%'.format(100 * p.get_height()/total) # percentage of each class of the category
x = p.get_x() + p.get_width() / 2 - 0.25 # width of the plot
y = p.get_y() + p.get_height() # hieght of the plot
ax.annotate(percentage, (x, y), size = 14) # annotate the percantage
plt.show() # show the plot
In [45]:
#Dependent_count
perc_on_bar(dataCC['Dependent_count'])
- Highest number of data points are for number of dependents 3, followed by 2
- Number of dependents 5 and no dependents have less data volume
In [46]:
#Total_Relationship_Count
perc_on_bar(dataCC['Total_Relationship_Count'])
- Most data points are for number of products 3
- Number of proucts 4,5 and 6 have almost same data volume
- Number of products 2,1 have less data volume
In [47]:
#Months_Inactive_12_mon
perc_on_bar(dataCC['Months_Inactive_12_mon'])
- Most data points are for number of months inactive 3 , followed by 2 and 1
- Data points are less for number of inactive months 4,5 and 6 which could mean more credit card usage
In [48]:
#Contacts_Count_12_mon
perc_on_bar(dataCC['Contacts_Count_12_mon'])
- Number of obsevations are high for number of contacts (number of times bank contacted the customer) 3, followed by 2
- Number of obsevations are low for higher number of contacts
In [49]:
#Education_Level
perc_on_bar(dataCC['Education_Level'])
- Number of observations are more for Graduate level followed by High School
- Unknown has 15% data points, this could be missing values and can be imputed later on
In [50]:
#Income_Category
perc_on_bar(dataCC['Income_Category'])
- Data points are highest for income category Less than 40K, followed by 40-60K
- Income category 120K + has less observations
- Unknown could be missing data and can be imputed later on
In [51]:
#Card_Category
perc_on_bar(dataCC['Card_Category'])
- 93% of observations are for card category Blue
- Silver has 5% and other categories have minimal data points
In [52]:
#Marital_Status
perc_on_bar(dataCC['Marital_Status'])
- Number of observations for Married is the highest, followed by Single
- Unknown marital status could be missing value and can be imputed later on
In [53]:
#Gender
perc_on_bar(dataCC['Gender'])
- Number of observations slightly more for Female, but not much difference
In [54]:
#Attrition_Flag
perc_on_bar(dataCC['Attrition_Flag'])
- This is our target variable, which is not balanced
- Class of interest for this problem statement is 'Attrited Customer' and that class has less data volume (16%)
EDA - Bivariate Analysis¶
- To see target variable Attrition_Flag in heatmap, converting it to numeric
- But for the rest of the analysis, it is more readable to keep it as category and hence will revert it back after heatmap and pairplot analysis is done
In [55]:
#Converting Attrition_Flag to 1 and 0 and chaging to integer
attrition = {'Existing Customer':0, 'Attrited Customer':1}
dataCC['Attrition_Flag']=dataCC['Attrition_Flag'].map(attrition)
dataCC['Attrition_Flag'] = dataCC['Attrition_Flag'].astype('int64')
In [56]:
#Heatmap to visualize the relationship between non-categorical variables
fig, ax = plt.subplots(figsize=(10,7));
sns.heatmap(dataCC.corr(),annot=True,linewidths=0.1,cmap='YlGnBu',fmt='.2f');
- Customer_Age and Months_on_book (period of relationship with the bank) have strong +ve correlation
- Credit_Limit has strong +ve correlation with Avg_Open_To_Buy (Avg open to buy credit limit). Both these features shows 100% correlation, becuase when credit limit goes up, the open to buy credit line also goes up, which is expected
- Credit_limit has -ve correlation with Avg_utilization_ratio, which shows with increase in credit limit the usage is not increasing
- Avg_utilization_ratio has -ve correlation with Avg_Open_to_buy (Avg open to buy credit limit), which shows with increase in the usage, open to buy credit line decreases
- Total_Revolving_Bal has strong +ve correlation with Avg_utilization_ratio, which is expected, with more usage of card, revolving balance goes up
- Total_Amt_Chng_Q4_Q1 (Change in transaction amount) has some +ve correlation with Total_Ct_Chng_Q4_Q1 (change in transaction count), which is expected, since both these features are inter-related
- Total_Trans_Amt (Total transaction amount) has some +ve correlation with Total_Trans_Ct (Total transaction count), which is expected, since both these features are inter-related
- Target variable "Attrition_Flag" is impacted by these features
- with increase in number of contacts b/w bank and customer, attrition increases
- with increase in number of inactive months, attrition increases
- with increase in number of products held by customer, attrition decreases
- with increase in total revolving balance, attrition decreases
- with increase in Change in Transaction Amount (Q4 over Q1), attrition decreases
- with increase in Change in Transaction Count (Q4 over Q1), attrition decreases
- with increase in Total Transaction Count, attrition decreases
- with increase in Total Transaction Amount, attrition decreases
- with increase in Average Card Utilization Ratio, attrition decreases
In [57]:
#Pairplot for the dataset
sns.pairplot(data=dataCC, hue='Attrition_Flag');
- Pairplot also shows the same relationsships b/w variables that we saw in heatmap
- It clearly shows there are very less data points for 'Attrited Customer'
In [58]:
#Converting Attrition_Flag back to original values and type category
attrition = {0:'Existing Customer', 1:'Attrited Customer'}
dataCC['Attrition_Flag']=dataCC['Attrition_Flag'].map(attrition)
dataCC['Attrition_Flag'] = dataCC['Attrition_Flag'].astype('category')
In [59]:
#Bivariate Analysis - let us look at how target variable (Attrition_Flag) is distributed amongst continuous variables
numeric_cols = ['Customer_Age','Months_on_book','Credit_Limit','Total_Revolving_Bal','Avg_Open_To_Buy','Total_Amt_Chng_Q4_Q1','Total_Trans_Amt','Total_Trans_Ct','Total_Ct_Chng_Q4_Q1','Avg_Utilization_Ratio','Attrition_Flag']
dataCC[numeric_cols].groupby(['Attrition_Flag']).mean()
Out[59]:
In [60]:
dataCC[numeric_cols].groupby(['Attrition_Flag']).median()
Out[60]:
- When we see mean and median of continuos variables with respect to the target variable, following features seem to have no major difference between the two categories (attrited and existing customers).
Customer_Age Months_on_book Credit_Limit (has slight diff, but not much) Avg_Open_To_Buy Total_Amt_Chng_Q4_Q1 Total_Ct_Chng_Q4_Q1 - It could be becuase these variables are not impacting the attrition, we will analyze further and see the feature importance after building the model to arrive at a final conclusion
- Total_Revolving_Bal (balance that carries over month to month) seems to be higher for existing customers, this could be because of the data imbalance b/w the two classes
- Total_Trans_Amt and Total_Trans_Ct seem to be higher for existing customers, this could be because of the data imbalance b/w the two classes
- Avg_Utilization_Ratio or usage of card seems to be high for existing customers,this could be because of the data imbalance b/w the two classes
In [61]:
#Bivariate Analysis - let us look at how target variable (Attrition_Flag) is distributed amongst categorical variables
pd.crosstab(dataCC['Dependent_count'],dataCC['Attrition_Flag'],normalize='index')
Out[61]:
- Dependent count does not seem to have an impact on target variable
- % of Attrited Vs Existing customers seem to be same across multiple values of dependent count
In [62]:
pd.crosstab(dataCC['Total_Relationship_Count'],dataCC['Attrition_Flag'],normalize='index')
Out[62]:
- This seem to have an impact on the target variable
- As number of products held by customers increase, % decrease in attrited customers and % of existing customers increase
In [63]:
pd.crosstab(dataCC['Months_Inactive_12_mon'],dataCC['Attrition_Flag'],normalize='index')
Out[63]:
- When number of inactive months is '0', the % of Attrited customers is almost same as existing customers
- When number of inactive months is more, there is some increase in % attrition
In [64]:
pd.crosstab(dataCC['Contacts_Count_12_mon'],dataCC['Attrition_Flag'],normalize='index')
Out[64]:
- This feature seem to have an impact on the target variable
- When the number of contacts increases, attrition rate also seems to increase. For the highest value of number of contacts, attrition is 100%
In [65]:
pd.crosstab(dataCC['Education_Level'],dataCC['Attrition_Flag'],normalize='index')
Out[65]:
- Education Level does not seem to have an impact on target variable
- % of Attrited Vs Existing customers seem to be same across different education levels
In [66]:
pd.crosstab(dataCC['Income_Category'],dataCC['Attrition_Flag'],normalize='index')
Out[66]:
- Income category does not seem to have an impact on target variable
- % of Attrited Vs Existing customers seem to be same across different income categories
In [67]:
pd.crosstab(dataCC['Card_Category'],dataCC['Attrition_Flag'],normalize='index')
Out[67]:
- Card category does not seem to have an impact on target variable
- % of Attrited Vs Existing customers seem to be almost same across different card categories
- Platinum shows slightly more attrition %
In [68]:
pd.crosstab(dataCC['Marital_Status'],dataCC['Attrition_Flag'],normalize='index')
Out[68]:
- Marital Status does not seem to have an impact on target variable
- % of Attrited Vs Existing customers seem to be almost same across different status
In [69]:
pd.crosstab(dataCC['Gender'],dataCC['Attrition_Flag'],normalize='index')
Out[69]:
- Gender does not seem to have an impact on target variable
- % of Attrited Vs Existing customers seem to be almost same across Male and Female
Let us explore the correlation between variables that we saw in heatmap¶
In [70]:
#Months_on_book Vs Customer_Age
sns.lineplot(y=dataCC.Months_on_book, x=dataCC.Customer_Age,hue=dataCC.Attrition_Flag);
- For both classes of target variable, there is some kind of linear relation b/w Age and Months on book
In [71]:
#Credit_Limit Vs Avg_Open_To_Buy
sns.lineplot(y=dataCC.Credit_Limit, x=dataCC.Avg_Open_To_Buy,hue=dataCC.Attrition_Flag);
- For both classes of target variable, there is clear linear relation b/w Credit Limit and Avg Open to Buy credit
In [72]:
#Credit_Limit Vs Avg_Utilization_Ratio
sns.lineplot(y=dataCC.Credit_Limit, x=dataCC.Avg_Utilization_Ratio,hue=dataCC.Attrition_Flag);
- For both classes of target variable, there is some kind of -ve correlation b/w Credit Limit and Avg Utilization Ratio
In [73]:
#Avg_Open_To_Buy Vs Avg_Utilization_Ratio
sns.lineplot(y=dataCC.Avg_Open_To_Buy, x=dataCC.Avg_Utilization_Ratio,hue=dataCC.Attrition_Flag);
- For both classes of target variable, there is some kind of -ve correlation b/w Avg Open to buy credit line and Avg Utilization Ratio
In [74]:
#Total_Revolving_Bal Vs Avg_Utilization_Ratio
sns.lineplot(y=dataCC.Total_Revolving_Bal, x=dataCC.Avg_Utilization_Ratio,hue=dataCC.Attrition_Flag);
- For both classes of target variable, there is some kind of linear relation b/w Avg_Utilization_Ratio and Total_Revolving_Bal
In [75]:
#Total_Trans_Amt Vs Total_Trans_Ct
sns.lineplot(y=dataCC.Total_Trans_Amt, x=dataCC.Total_Trans_Ct,hue=dataCC.Attrition_Flag);
- For both classes of target variable, there is some kind of linear relation b/w Total_Trans_Amt and Total_Trans_Ct
Attrition_Flag Vs continuous variables¶
In [76]:
#Bivariate plot for numeric variables and target variable
cols = dataCC[
[
"Customer_Age",
"Months_on_book",
"Credit_Limit",
"Total_Revolving_Bal",
"Avg_Open_To_Buy",
"Total_Amt_Chng_Q4_Q1",
"Total_Trans_Amt",
"Total_Trans_Ct",
"Total_Ct_Chng_Q4_Q1",
"Avg_Utilization_Ratio"
]
].columns.tolist()
plt.figure(figsize=(10, 20))
for i, variable in enumerate(cols):
plt.subplot(5, 2, i + 1)
sns.boxplot(dataCC["Attrition_Flag"], dataCC[variable])
plt.tight_layout()
plt.title(variable)
plt.show()
- Similar to the analysis done using median and mean, the boxplots also show that the target variable is not much impacted by Customer Age,Months on book and Avg open to buy credit
- There is very slight difference b/w two catefories for Credit limit, Change in total amount and Change in total count
- For existing customers, median and 75% are higher for total revolving balance
- Total transaction amount and Total transaction count are higher for existing customers
- Utilization Ratio Median and 75% are higher for existing customers
In [77]:
#Function to get count plot with hue, that shows % among each class
#Shows crosstab again for better readability and to show % within class and different classes altogether
def perc_on_bar_with_hue(x):
tab1 = pd.crosstab(dataCC[x],dataCC['Attrition_Flag'],margins=True)
print(tab1)
print('-'*80)
df1 = dataCC.groupby(x)['Attrition_Flag'].value_counts(normalize=True)
df1 = df1.mul(100)
df1 = df1.rename('percent').reset_index()
g = sns.catplot(x=x,y='percent',hue='Attrition_Flag',kind='bar',data=df1,aspect=2.0)
g.ax.set_ylim(0,100)
for p in g.ax.patches:
#print(p)
height = p.get_height()
if np.isnan(height):
height = 0
txt = str(round(height,2)) + '%'
txt_x = p.get_x()
txt_y = height
g.ax.text(txt_x,txt_y,txt)
In [78]:
#Dependent_count Vs Attrition_Flag
perc_on_bar_with_hue('Dependent_count')
- % of existing customers Vs attrited customers seem to be almost same accross multiple values of dependent count
- So dependent count does not look like it is impacting the target variable
In [79]:
#Total_Relationship_Count Vs Attrition_Flag
perc_on_bar_with_hue('Total_Relationship_Count')
- For higher number of products held by customers, attrition % goes down and existing customer % goes up
In [80]:
#Months_Inactive_12_mon Vs Attrition_Flag
perc_on_bar_with_hue('Months_Inactive_12_mon')
- When number of inactive months is '0', the % of Attrited customers is almost same as existing customers
- When number of inactive months is more, there is % attrition increases first and then after a point it comes down
In [81]:
#Contacts_Count_12_mon Vs Attrition_Flag
perc_on_bar_with_hue('Contacts_Count_12_mon')
- When the number of contacts increases, attrition rate also seems to increase. For the highest value of number of contacts, attrition is 100%, may be these are the customers need to be contacted by the bank multiple times for payment
In [82]:
#Education_Level Vs Attrition_Flag
perc_on_bar_with_hue('Education_Level')
- Education Level does not seem to have an impact on target variable
- % of Attrited Vs Existing customers seem to be same across different education levels, except for Doctorate, that has a slightly higher % attrition
In [83]:
#Income_Category Vs Attrition_Flag
perc_on_bar_with_hue('Income_Category')
- % of Attrited Vs Existing customers seem to be almost same across different income categories
In [84]:
#Card_Category Vs Attrition_Flag
perc_on_bar_with_hue('Card_Category')
- Platinum shows bit more % in attrited customers
- Other than that, % of attrited customers Vs existing customers seem to be same accross card categories
In [85]:
#Marital_Status Vs Attrition_Flag
perc_on_bar_with_hue('Marital_Status')
- % of Attrited Vs Existing customers seem to be almost same across different status
In [86]:
#Gender Vs Attrition_Flag
perc_on_bar_with_hue('Gender')
% of Attrited Vs Existing customers seem to be almost same across Male and Female (there is very slight % increase in Female)
Insights based on EDA¶
Key meaningful observations on the relationship between variables¶
- In terms of attrition %, customers with more total revolving balance tend to keep using the bank credit card, hence less attrition rate
- In terms of attrition %, customers with more total transaction amount in last 12 months seem to keep using the bank credit card, hence less attrition rate
- In terms of attrition %, customers with more total transaction count in last 12 months seem to keep using the bank credit card, hence less attrition rate
- In terms of attrition %, customers with more card utilization seem to keep using the bank credit card, hence less attrition rate
- Attrition % seems to be lower for customers having more number of products offered by the bank
- Attrition % seems to increase when the card is inactive for longer period of time
- For customers who need to be contacted by the bank multiple times, they tend to become attrited customers
- With increase in Change in Transaction Amount (Q4 over Q1), attrition decreases
- With increase in Change in Transaction Count (Q4 over Q1), attrition decreases
- Platinum card shows bit more attrition % comapred to other types of cards
- Attrition % seems to be little bit more in Females compared to Males, not a significant difference though
- Customer Age and Period of relationship with the bank are two features that are positively correlated
- Credit Limit and Avg open to buy credit line are positively correlated, could be because when credit limit goes up more amount is open for usage
- Credit Limit and Avg utilization are negatively correlated, that means with increase in credit limit, usage is not increasing
- Avg utilization and Avg open to buy credit line are negatively correlated, which means with more usage the available credit line decreases
- Avg utilization and Total revolving balance are positively correlated, which make sense, when there is more usage, balance that carries over month to month also increases
Data Pre-processing¶
- CLIENTNUM column removed since that is just an ID column and not going to add value to the model
- There was an empty column (Unnamed) in the dataset, that is removed
- Missing data (unknown) present in columns - Education_Level, Income_Category and Marital_Status, we will handle these missing values in this data pre-processing step
- Outliers are present in some of the columns and they were explained in the univariate analysis. Since those values are not standing out from the rest of the data, they are not removed
- Target variable (Attrition_Flag) is a categorical featured with just two values and hence can be mapped to 1 and 0
- Since Credit limit shows 100% correlation with Avg_open_to_buy, one of this features can be dropped
- Since Total_Trans_Ct shows 81% correlation with Total_Trans_Amt, one of this features can be dropped
In [87]:
#Change target variable (Attrition_Flag) to 1 and 0
#Attrition_Flag: Internal event (customer activity) variable - if the account is closed then 1 else 0
attrition = {'Existing Customer':0, 'Attrited Customer':1}
dataCC['Attrition_Flag']=dataCC['Attrition_Flag'].map(attrition)
In [88]:
#Checking columns that contain zeros
for col in dataCC.drop('Attrition_Flag',axis=1).columns:
if (dataCC[col] == 0).any() == True:
print('{} has one or more zeros'.format(col))
- The above 7 columns have zeros, but they are features that can have zeros in real world scenarios, hence not imputing them
In [89]:
#For KNN to work, the columns for which we do KNN imputation need to be converted to numeric
#After KNN imputation for missing values is done, reverse mapping can be done to restore original values
education = {'Uneducated':1, 'High School':2, 'College':3, 'Graduate':4, 'Post-Graduate':5, 'Doctorate':6}
dataCC['Education_Level']=dataCC['Education_Level'].map(education)
income = {'Less than $40K':1, '$40K - $60K':2, '$60K - $80K':3, '$80K - $120K':4, '$120K +':5}
dataCC['Income_Category']=dataCC['Income_Category'].map(income)
marital_status = {'Married':1,'Single':2, 'Divorced':3}
dataCC['Marital_Status']=dataCC['Marital_Status'].map(marital_status)
In [90]:
#Checking nulls
dataCC.isnull().sum().sort_values(ascending=False)
Out[90]:
In [91]:
# Separating target variable and other variables
X = dataCC.drop(columns='Attrition_Flag',axis=1)
Y = dataCC['Attrition_Flag'].astype('category')
In [92]:
#Dropping Avg_Open_To_Buy, since Credit limit shows 100% correlation with Avg_open_to_buy
X.drop('Avg_Open_To_Buy',axis=1,inplace=True)
In [93]:
#Dropping Total_Trans_Ct, since Total_Trans_Ct shows 81% correlation with Total_Trans_Amt
X.drop('Total_Trans_Ct',axis=1,inplace=True)
In [94]:
# Splitting the data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(
X, Y, test_size=0.30, random_state=1, stratify=Y
)
print(X_train.shape, X_test.shape)
In [95]:
#checking target variable imbalance in train and test set
print('Existing customer % in training dataset is: {}'.format((y_train == 0).sum()/y_train.count()) )
print('Attrited customer % in training dataset is: {}'.format((y_train == 1).sum()/y_train.count()) )
print('Existing customer % in test dataset is: {}'.format((y_test == 0).sum()/y_test.count()) )
print('Attrited customer % in test dataset is: {}'.format((y_test == 1).sum()/y_test.count()) )
- Target variable is 84% Existing customer class and 16% Attrited customer class
Missing-Value Treatment¶
In [96]:
imputer = KNNImputer(n_neighbors=5)
In [97]:
reqd_col_for_impute = ['Education_Level','Income_Category','Marital_Status']
In [98]:
#Fit and transform the train data
X_train[reqd_col_for_impute]=imputer.fit_transform(X_train[reqd_col_for_impute])
#Transform the test data
X_test[reqd_col_for_impute]=imputer.transform(X_test[reqd_col_for_impute])
In [99]:
#Checking that no column has missing values in train or test sets
print(X_train.isna().sum())
print('-'*30)
print(X_test.isna().sum())
- All missing values are imputed
- Now we can inverse map the encoded values in Education_Level,Income_Category and Marital_Status
In [100]:
## Function to inverse the encoding
def inverse_mapping(x,y):
inv_dict = {v: k for k, v in x.items()}
X_train[y] = np.round(X_train[y]).map(inv_dict).astype('category')
X_test[y] = np.round(X_test[y]).map(inv_dict).astype('category')
In [101]:
inverse_mapping(education,'Education_Level')
inverse_mapping(income,'Income_Category')
inverse_mapping(marital_status,'Marital_Status')
In [102]:
#Checking inverse mapped values/categories on training dataset
for i in reqd_col_for_impute:
print(X_train[i].value_counts())
print('*'*30)
In [103]:
#Checking inverse mapped values/categories on test dataset
for i in reqd_col_for_impute:
print(X_test[i].value_counts())
print('*'*30)
- Inverse mapping returned original labels
Encoding categorical varaibles¶
In [104]:
X_train=pd.get_dummies(X_train,drop_first=True)
X_test=pd.get_dummies(X_test,drop_first=True)
print(X_train.shape, X_test.shape)
- After encoding there are 27 columns
In [105]:
X_train.info()
Model Building¶
Model performance evaluation - right metric to use¶
- The objective is to find out if a customer is going to leave credit card services or not. Because customers leaving credit card services would lead bank to loss, so the bank wants to analyze the data of customers and identify the customers who will leave their credit card services
Here are the wrong predictions that the model can make¶
- Predicting a customer will leave credit card services and the customer does not leave
- Predicting a customer will NOT leave credit card services and the customer leaves
Which prediction is more important¶
- In the first scenario, the model predicts customer will leave credit card services, so the bank takes effort to retain them, which causes the bank more resources and money
- In the second scenario, the model predicts customer will NOT leave credit card services, so the bank is not going to take any effort to retain those customers, but in real world scenario the customer leaves the serives, which causes the bank huge loss
Which is the right metric¶
- The first wrong prediction type is FP (False Positive) and the second wrong prediction type is FN(False Negative)
- Since the bank wants to avoid losses due to customers leaving credit card services, we need to focus on reducing the false negatives, recall is the right metric in this case, greater the Recall lesser the chances of false negatives
- We will focus more on Recall, but also keep an eye on precision, which is the metric if our focus is on reducing false positives, in this case reducing the resources
How to interpret confusion matrix for the given problem¶
True Positives:
- Reality: A customer leaves credit card services
- Model predicted: The customer will leave credit card services
- Outcome: The model is good
True Negatives:
- Reality: A customer does not leave credit card services
- Model predicted: The customer will NOT leave credit card services
- Outcome: The model is good
False Positives:
- Reality: A customer does not leave credit card services
- Model predicted: The customer will leave credit card services
- Outcome: Bank will be spending more on resources trying to reatin the customers who will not leave credit card services
False Negatives:
- Reality: A customer leaves credit card services
- Model predicted: The customer will NOT leave credit card services
- Outcome: Bank will face a huge loss, since they will not focus on retaining the potential customers who will leave credit card services
Model evaluation¶
- It is recommended to split the dataset into training,evaluation and testing and fit the model on training set, perform hyper parameter tuning on evaluation set and after finding the right set of parameters check if the model is underfit, overfit or right fit on the test set
- Here Stratified K-Folds cross-validator is used, which will split data in train/validation sets. Split dataset into k consecutive folds keeping distribution of both classes in each fold same as the target variable. Each fold is then used once as a validation while the k - 1 remaining folds form the training set.
In [106]:
## Function to calculate different metric scores of the model - Accuracy, Recall and Precision
def get_metrics_score(model,train,test,train_y,test_y,flag=True):
'''
model : classifier to predict values of X
'''
# defining an empty list to store train and test results
score_list=[]
pred_train = model.predict(train)
pred_test = model.predict(test)
train_acc = model.score(train,train_y)
test_acc = model.score(test,test_y)
train_recall = metrics.recall_score(train_y,pred_train)
test_recall = metrics.recall_score(test_y,pred_test)
train_precision = metrics.precision_score(train_y,pred_train)
test_precision = metrics.precision_score(test_y,pred_test)
score_list.extend((train_acc,test_acc,train_recall,test_recall,train_precision,test_precision))
# If the flag is set to True then only the following print statements will be dispayed. The default value is set to True.
if flag == True:
print("Accuracy on training set : ",model.score(train,train_y))
print("Accuracy on test set : ",model.score(test,test_y))
print("Recall on training set : ",metrics.recall_score(train_y,pred_train))
print("Recall on test set : ",metrics.recall_score(test_y,pred_test))
print("Precision on training set : ",metrics.precision_score(train_y,pred_train))
print("Precision on test set : ",metrics.precision_score(test_y,pred_test))
return score_list # returning the list with train and test scores
In [107]:
#Function to display confusion matrix
def make_confusion_matrix(model,y_actual,labels=[1, 0]):
'''
model : classifier to predict values of X
y_actual : ground truth
'''
y_predict = model.predict(X_test)
cm=metrics.confusion_matrix( y_actual, y_predict, labels=[0, 1])
df_cm = pd.DataFrame(cm, index = [i for i in ["Actual - No","Actual - Yes"]],
columns = [i for i in ['Predicted - No','Predicted - Yes']])
group_counts = ["{0:0.0f}".format(value) for value in
cm.flatten()]
group_percentages = ["{0:.2%}".format(value) for value in
cm.flatten()/np.sum(cm)]
labels = [f"{v1}\n{v2}" for v1, v2 in
zip(group_counts,group_percentages)]
labels = np.asarray(labels).reshape(2,2)
plt.figure(figsize = (10,7))
sns.heatmap(df_cm, annot=labels,fmt='')
plt.ylabel('True label')
plt.xlabel('Predicted label')
Model building - Logistic Regression¶
In [108]:
lr = LogisticRegression(random_state=1)
lr.fit(X_train,y_train)
Out[108]:
Evaluating model performance by using KFold and cross_val_score¶
In [109]:
scoring='recall'
kfold=StratifiedKFold(n_splits=5,shuffle=True,random_state=1) #Setting number of splits equal to 5
cv_result_bfr=cross_val_score(estimator=lr, X=X_train, y=y_train, scoring=scoring, cv=kfold)
print(cv_result_bfr)
#Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_bfr)
plt.show()
- Performance on training set varies between 0.31 and 0.40 recall (boxplot is showing a distribution with one outlier)
In [110]:
#Checking the performance on test dataset
#Calculating different metrics
get_metrics_score(lr,X_train,X_test,y_train,y_test)
# creating confusion matrix
make_confusion_matrix(lr,y_test)
- Logistic Regression has given a generalized performance on training and test set.
- Recall is very low, we will try oversampling and undersampling to see if the model performance can be improved.
- From confusion matrix, false negative is 11% and our focus should be reducing that, since these cause a huge loss for the bank as explained in "how to interpret confusion matrix"
- True positives which are correct positive predictions is only 5%
Oversampling train data using SMOTE¶
In [111]:
print("Before UpSampling, counts of label 'Existing': {}".format(sum(y_train==0)))
print("Before UpSampling, counts of label 'Attrited': {} \n".format(sum(y_train==1)))
sm = SMOTE(sampling_strategy = 1 ,k_neighbors = 5, random_state=1) #Synthetic Minority Over Sampling Technique
X_train_over, y_train_over = sm.fit_resample(X_train, y_train)
print("After UpSampling, counts of label 'Existing': {}".format(sum(y_train_over==0)))
print("After UpSampling, counts of label 'Attrited': {} \n".format(sum(y_train_over==1)))
print('After UpSampling, the shape of train_X: {}'.format(X_train_over.shape))
print('After UpSampling, the shape of train_y: {} \n'.format(y_train_over.shape))
Logistic Regression on oversampled data¶
In [112]:
log_reg_over = LogisticRegression(random_state = 1)
# Training the basic logistic regression model with training set
log_reg_over.fit(X_train_over,y_train_over)
Out[112]:
Evaluating model performance by using KFold and cross_val_score¶
In [113]:
scoring='recall'
kfold=StratifiedKFold(n_splits=5,shuffle=True,random_state=1) #Setting number of splits equal to 5
cv_result_over=cross_val_score(estimator=log_reg_over, X=X_train_over, y=y_train_over, scoring=scoring, cv=kfold)
print(cv_result_over)
#Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_over)
plt.show()
- Performance on training set varies between 0.72 and 0.77 recall, which is an improvement from the base model and it is pretty consistent
In [114]:
#Checking the performance on test dataset
#Calculating different metrics
get_metrics_score(log_reg_over,X_train_over,X_test,y_train_over,y_test)
# creating confusion matrix
make_confusion_matrix(log_reg_over,y_test)
Performance on the training set improved but the model is showing a clear difference b/w train and test scores
False negative came down to 5.6%, compared to previous model, which is good
True positives which are correct positive predictions increased to 10%, compared to previous model, which is also good
Model is overfitting, train/test scores have a clear difference.
We can try both these options:
a) Regularization to see if overfitting can be reduced
b) Undersampling the train set to handle the imbalance between classes and check the model performance.
Regularization¶
In [115]:
# Choose the type of classifier.
#LogisticRegression has a parameter 'penalty', default values is l2. SAGA solver supports l1 penalty.
#l1 is Lasso Regression and l2 is Ridge Regression
#Since we use SAGA solver, we are implementing Lasso
lr_estimator = LogisticRegression(random_state=1,solver='saga')
# Grid of parameters to choose from
parameters = {'C': np.arange(0.1,1.1,0.1)}
# Run the grid search
grid_obj = GridSearchCV(lr_estimator, parameters, scoring='recall')
grid_obj = grid_obj.fit(X_train_over, y_train_over)
# Set the clf to the best combination of parameters
lr_estimator = grid_obj.best_estimator_
# Fit the best algorithm to the data.
lr_estimator.fit(X_train_over, y_train_over)
Out[115]:
In [116]:
#Calculating different metrics
get_metrics_score(lr_estimator,X_train_over,X_test,y_train_over,y_test)
# creating confusion matrix
make_confusion_matrix(lr_estimator,y_test)
Over-fitting has reduced to a good extent, test recall is lower compared to previous model, but train and test recall are close now
False negative has come down to 7%, compared to base model
True positives which are correct positive predictions increased to 9%, compared to base model
We can try downsampling now
Undersampling train data using RandomUnderSampler¶
In [117]:
rus = RandomUnderSampler(random_state = 1)
X_train_un, y_train_un = rus.fit_resample(X_train, y_train)
In [118]:
print("Before Under Sampling, counts of label 'Existing': {}".format(sum(y_train==0)))
print("Before Under Sampling, counts of label 'Attrited': {} \n".format(sum(y_train==1)))
print("After Under Sampling, counts of label 'Existing': {}".format(sum(y_train_un==0)))
print("After Under Sampling, counts of label 'Attrited': {} \n".format(sum(y_train_un==1)))
print('After Under Sampling, the shape of train_X: {}'.format(X_train_un.shape))
print('After Under Sampling, the shape of train_y: {} \n'.format(y_train_un.shape))
Logistic Regression on undersampled data¶
In [119]:
log_reg_under = LogisticRegression(random_state = 1)
log_reg_under.fit(X_train_un,y_train_un )
Out[119]:
In [120]:
scoring='recall'
kfold=StratifiedKFold(n_splits=5,shuffle=True,random_state=1) #Setting number of splits equal to 5
cv_result_under=cross_val_score(estimator=log_reg_under, X=X_train_un, y=y_train_un, scoring=scoring, cv=kfold)
print(cv_result_under)
#Plotting boxplots for CV scores of model defined above
plt.boxplot(cv_result_under)
plt.show()
- Performance of model on training set varies between 0.72 to 0.75, which is an improvement from the initial model(without oversampling)
In [121]:
#Checking the performance on test dataset
#Calculating different metrics
get_metrics_score(log_reg_under,X_train_un,X_test,y_train_un,y_test)
# creating confusion matrix
make_confusion_matrix(log_reg_under,y_test)
- Model has given a generalized performance on training and test set.
- Model performance(recall) has improved using downsampling, it is better than the initial model and the one with oversampling
- Looking at the confusion matrix, False Negative has come down really well, which is good
- True positives which are correct positive predictions increased to 11%, which is also very good for the model
Finding the coefficients from logistic regression¶
In [122]:
log_odds = log_reg_under.coef_[0]
df = pd.DataFrame(log_odds, X_train_un.columns, columns=['coef'])
df.reindex(df.coef.abs().sort_values(ascending=False).index)
Out[122]:
Converting coefficients to odds¶
In [123]:
odds = np.exp(log_reg_under.coef_[0])-1
pd.set_option('display.max_rows',None)
df=pd.DataFrame(odds, X_train_un.columns, columns=['odds'])
df.reindex(df.odds.abs().sort_values(ascending=False).index)
Out[123]:
Concluding findings from logistic regression¶
Contacts_Count_12_mon: For one-unit increase in the number of contacts in last 12 months, we expect to see about a 52% increase in the odds of a customer leaving credit card services
Months_Inactive_12_mon: For one-unit increase in the number of inactive months in last 12 months, we expect to see about a 35% increase in the odds of a customer leaving credit card services
Total_Relationship_Count: For one-unit increase in the number of products owned by a customer, we expect to see about a 29% decrease in the odds of a customer leaving credit card services
Dependent_count: For one-unit increase in the number of dependents, we expect to see about a 12% increase in the odds of a customer leaving credit card services
Total_Ct_Chng_Q4_Q1: For one-unit increase in the number of transactions between Q1 and Q4, we expect to see about a 11% decrease in the odds of a customer leaving credit card services
All other features can be interpretted in the same way
We will try other models and see the feature importance
Model building - Bagging and Boosting¶
- Building Decision tree, random forest, bagging classifier, Xgboost, AdaBoost, and gradient boosting models using pipeline
Stratified K-Folds cross-validator is used, which will split data in train/validation sets. Split dataset into k consecutive folds keeping distribution of both classes in each fold same as the target variable. Each fold is then used once as a validation while the k - 1 remaining folds form the training set.
This is better than splitting the data into just 3 sets - train,validation and test
Since I want to apply StandardScaler() on the dataset and fit the models, I am using pipeline() to do both together
DecisionTreeClassifier¶
In [124]:
results = [] # Empty list to store all model's CV scores
names = [] # Empty list to store name of the models
In [125]:
dtree = Pipeline(
steps=[
("scaler", StandardScaler()),
("d_tree", DecisionTreeClassifier(criterion='gini',class_weight={0:0.16,1:0.84},random_state=1))
]
)
dtree.fit(X_train,y_train)
Out[125]:
Evaluating model performance by using KFold and cross_val_score¶
In [126]:
scoring='recall'
kfold=StratifiedKFold(n_splits=5,shuffle=True,random_state=1) #Setting number of splits equal to 5
cv_result_dtree=cross_val_score(estimator=dtree, X=X_train, y=y_train, scoring=scoring, cv=kfold)
print(cv_result_dtree)
results.append(cv_result_dtree)
names.append('DTREE')
print("{}: {}".format('Mean recall score for DecisionTreeClassifier is', cv_result_dtree.mean() * 100))
- Performance on training set varies between 0.71 and 0.77 recall.
In [127]:
#Checking the performance on test dataset
#Calculating different metrics
get_metrics_score(dtree,X_train,X_test,y_train,y_test)
# creating confusion matrix
make_confusion_matrix(dtree,y_test)
- With k-fold cross validation on training set the avg recall score is 73.3%
- But when we fit the model on test set, it clearly shows over-fitting
- Looking at the confusion matrix, False Negative is low, which is good, as explained in "how to interpret confusion matrix"
- True positives which are correct positive predictions increased to 12%, which is also very good for the model
Feature importance - DecisionTree¶
In [128]:
feature_names = X_train.columns
importances = dtree[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_trans_amt, total_revolving_bal are the 2 most important features impacting attrition as per decision tree model
- Decision Tree model is over-fitting on training data
- False Negative is low, which is good, since that indicate lower loss for the bank, since the model is making less mistakes in making -ve predictions (model predicts customer will NOT leave, but they actually leave)
RandomForest Classifier¶
In [129]:
rf = Pipeline(
steps=[
("scaler", StandardScaler()),
("random_forest", RandomForestClassifier(random_state=1))
]
)
rf.fit(X_train,y_train)
Out[129]:
Evaluating model performance by using KFold and cross_val_score¶
In [130]:
scoring='recall'
kfold=StratifiedKFold(n_splits=5,shuffle=True,random_state=1) #Setting number of splits equal to 5
cv_result_rf=cross_val_score(estimator=rf, X=X_train, y=y_train, scoring=scoring, cv=kfold)
print(cv_result_rf)
results.append(cv_result_rf)
names.append('RF')
print("{}: {}".format('Mean recall score for RandomForestClassifier is', cv_result_rf.mean() * 100))
- Performance on training set varies between 0.67 and 0.76 recall.
In [131]:
#Checking the performance on test dataset
#Calculating different metrics
get_metrics_score(rf,X_train,X_test,y_train,y_test)
# creating confusion matrix
make_confusion_matrix(rf,y_test)
- With k-fold cross validation on training set the avg recall score is 69.9%
- But when we fit the model on test set, it again shows over-fitting
- Looking at the confusion matrix, False Negative is bit higher than decision tree
- True positives which are correct positive predictions also is slightly lower than decision tree
Feature importance - Random Forest¶
In [132]:
feature_names = X_train.columns
importances = rf[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_trans_amt, Change in Transaction Count (Q4 over Q1) are the 2 most important features impacting attrition as per random forest model
- This model is over-fitting on training data
- False Negative is bit higher than Decision Tree and True positive is bit lower compared to decision tree
BaggingClassifier¶
In [133]:
bc = Pipeline(
steps=[
("scaler", StandardScaler()),
("bagging", BaggingClassifier(random_state=1))
]
)
bc.fit(X_train,y_train)
Out[133]:
Evaluating model performance by using KFold and cross_val_score¶
In [134]:
scoring='recall'
kfold=StratifiedKFold(n_splits=5,shuffle=True,random_state=1) #Setting number of splits equal to 5
cv_result_bc=cross_val_score(estimator=bc, X=X_train, y=y_train, scoring=scoring, cv=kfold)
print(cv_result_bc)
results.append(cv_result_bc)
names.append('BC')
print("{}: {}".format('Mean recall score for BaggingClassifier is', cv_result_bc.mean() * 100))
- Performance on training set varies between 0.71 and 0.82 recall.
In [135]:
#Checking the performance on test dataset
#Calculating different metrics
get_metrics_score(bc,X_train,X_test,y_train,y_test)
# creating confusion matrix
make_confusion_matrix(bc,y_test)
- With k-fold cross validation on training set the avg recall score is 74.18%
- Test recall score is good
- But when we fit the model on test set, it shows clear difference b/w train and test scores
- Looking at the confusion matrix, False Negative is low, which is good
- True positives which are correct positive predictions is 12.44%, which is also very good for the model
Bagging classifier has no feature importance attribute¶
XGBClassifier¶
In [136]:
xgb = Pipeline(
steps=[
("scaler", StandardScaler()),
("xgboost", XGBClassifier(random_state=1,eval_metric='logloss'))
]
)
xgb.fit(X_train,y_train)
Out[136]:
Evaluating model performance by using KFold and cross_val_score¶
In [137]:
scoring='recall'
kfold=StratifiedKFold(n_splits=5,shuffle=True,random_state=1) #Setting number of splits equal to 5
cv_result_xgb=cross_val_score(estimator=xgb, X=X_train, y=y_train, scoring=scoring, cv=kfold)
print(cv_result_xgb)
results.append(cv_result_xgb)
names.append('XGB')
print("{}: {}".format('Mean recall score for XGBClassifier is', cv_result_xgb.mean() * 100))
- Performance on training set varies between 0.79 and 0.87 recall.
In [138]:
#Checking the performance on test dataset
#Calculating different metrics
get_metrics_score(xgb,X_train,X_test,y_train,y_test)
# creating confusion matrix
make_confusion_matrix(xgb,y_test)
- With k-fold cross validation on training set the avg recall score is 83%
- But when we fit the model on test set, it is over-fitting
- Looking at the confusion matrix, False Negative is low, which is good
- True positives which are correct positive predictions is 14%, which is also very good for the model
Feature importance - XGBoost¶
In [139]:
feature_names = X_train.columns
importances = xgb[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Revolving_Bal, Total_Relationship_Count are the 2 most important features impacting attrition from XGBoost
- This model is over-fitting on training data
- False Negative is low, which is good and True positive is improved a bit
Adaboost¶
In [140]:
abc = Pipeline(
steps=[
("scaler", StandardScaler()),
("adaboost", AdaBoostClassifier(random_state=1))
]
)
abc.fit(X_train,y_train)
Out[140]:
Evaluating model performance by using KFold and cross_val_score¶
In [141]:
scoring='recall'
kfold=StratifiedKFold(n_splits=5,shuffle=True,random_state=1) #Setting number of splits equal to 5
cv_result_abc=cross_val_score(estimator=abc, X=X_train, y=y_train, scoring=scoring, cv=kfold)
print(cv_result_abc)
results.append(cv_result_abc)
names.append('ABC')
print("{}: {}".format('Mean recall score for AdaboostClassifier is', cv_result_abc.mean() * 100))
- Performance on training set varies between 0.69 and 0.77 recall.
In [142]:
#Checking the performance on test dataset
#Calculating different metrics
get_metrics_score(abc,X_train,X_test,y_train,y_test)
# creating confusion matrix
make_confusion_matrix(abc,y_test)
- With k-fold cross validation on training set the avg recall score is 73.6%
- When we fit the model on test set, it is not over-fitting and gives good scores (test recall score 73% is almost same as avg CV score)
- Looking at the confusion matrix, False Negative is bit higher than xgboost
- True positives which are correct positive predictions is 12%, bit lower than xgboost
Feature importance - Adaboost¶
In [143]:
feature_names = X_train.columns
importances = abc[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Trans_Amt, Change in Transaction Amount (Q4 over Q1) are the 2 most important features impacting attrition as per AdaBoost model
- This model is not over-fitting on training data and gives good scores
- False Negative is bit higher than xgboost, and True positive is bit lower than xgboost
GradientBoostingClassifier¶
In [144]:
gbm = Pipeline(
steps=[
("scaler", StandardScaler()),
("gradient_boosting", GradientBoostingClassifier(random_state=1))
]
)
gbm.fit(X_train,y_train)
Out[144]:
Evaluating model performance by using KFold and cross_val_score¶
In [145]:
scoring='recall'
kfold=StratifiedKFold(n_splits=5,shuffle=True,random_state=1) #Setting number of splits equal to 5
cv_result_gbm=cross_val_score(estimator=gbm, X=X_train, y=y_train, scoring=scoring, cv=kfold)
print(cv_result_gbm)
results.append(cv_result_gbm)
names.append('GBM')
print("{}: {}".format('Mean recall score for GradientboostClassifier is', cv_result_gbm.mean() * 100))
- Performance on training set varies between 0.71 and 0.82 recall.
In [146]:
#Checking the performance on test dataset
#Calculating different metrics
get_metrics_score(gbm,X_train,X_test,y_train,y_test)
# creating confusion matrix
make_confusion_matrix(gbm,y_test)
- With k-fold cross validation on training set the avg recall score is 76.11%
- When we fit the model on test set, it is not over-fitting and gives good scores
- Looking at the confusion matrix, False Negative is lower and it is good
- True positives which are correct positive predictions is 13%, which is also very good for the model
Feature importance - GradientBoost¶
In [147]:
feature_names = X_train.columns
importances = gbm[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Trans_Amt, Total_Revolving_bal are the 2 most important features impacting attrition from GradientBoost
- This model is not over-fitting on training data and gives good scores
- False Negative is low, which is good
CV scores for all models in a boxplot with avg score for each model¶
In [148]:
# Plotting boxplots for CV scores of all models defined above
fig = plt.figure(figsize=(10, 7))
fig.suptitle("Algorithm Comparison")
ax = fig.add_subplot(111)
dfResults = pd.DataFrame(results,index=names,columns=['CV_Score1','CV_Score2','CV_Score3','CV_Score4','CV_Score5'])
#sorting the CV results for each model (row wise) and showing the max CV score at the end (last column) and avg score
a = dfResults.values
a.sort(axis=1)
dfResults1 = pd.DataFrame(a, dfResults.index, dfResults.columns)
dfResults1['AVG_Score'] = dfResults1.mean(axis=1)
print(dfResults1)
#print(results)
plt.boxplot(results)
ax.set_xticklabels(names)
plt.show()
- We can see that XGBoost is giving the highest cross validated recall followed by GradientBoost
- We will tune all these models using GridSearch and RandomSearch and see if the performance improves.
Hyperparameter Tuning Using GridSearch¶
Using pipelines with StandardScaler and each of the above model to tune the model using GridSearchCV and RandomizedSearchCV.
Since I have already used Pipeline() method to create pipelines in the above code, I am using make_pipeline() function here
Decision Tree - GridSearchCV¶
In [149]:
%%time
# Creating pipeline
pipe = make_pipeline(StandardScaler(), DecisionTreeClassifier(criterion='gini',class_weight={0:0.16,1:0.84},random_state=1))
# Parameter grid to pass in GridSearchCV
param_grid = {
'decisiontreeclassifier__max_depth': np.arange(1,10),
'decisiontreeclassifier__min_samples_leaf': np.arange(1,10,2),
'decisiontreeclassifier__max_leaf_nodes' : np.arange(1,10,2),
'decisiontreeclassifier__min_impurity_decrease': [0.0001,0.001,0.01],
'decisiontreeclassifier__max_features': np.arange(0.2, 0.7, 0.2)
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
# Calling GridSearchCV
grid_cv = GridSearchCV(estimator=pipe, param_grid=param_grid, scoring=scorer, cv=5)
# Fitting parameters in GridSeachCV
grid_cv.fit(X_train, y_train)
print(
"Best Parameters:{} \nScore: {}".format(grid_cv.best_params_, grid_cv.best_score_)
)
In [150]:
# Creating new pipeline with best parameters
dtree_tuned1 = make_pipeline(
StandardScaler(),
DecisionTreeClassifier(
criterion='gini',class_weight={0:0.16,1:0.84},random_state=1,
max_depth=4,
max_leaf_nodes=9,
min_samples_leaf=1,
min_impurity_decrease=0.0001,
max_features=0.4
)
)
# Fit the model on training data
dtree_tuned1.fit(X_train, y_train)
Out[150]:
In [151]:
# Calculating different metrics
get_metrics_score(dtree_tuned1,X_train,X_test,y_train,y_test)
# Creating confusion matrix
make_confusion_matrix(dtree_tuned1, y_test)
- Recall on training set is 91% and recall on test set is 91%
- Train and test recall scores are very close and good
- Precision and accuracy scores are not good and can be improved
- False Negative is very low which is really good
- True positives which are correct positive predictions is 15%, which is also very good for the model
Feature importance - Decision Tree using GridSearch¶
In [152]:
feature_names = X_train.columns
importances = dtree_tuned1[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Trans_Amt,Total_Revolving_Bal are the 2 most important features impacting attrition as per Decision Tree using GridSearch
- Model gives good train and test scores, but accuracy and precision are low
- False Negative is low and True positive is high, which is good
Random Forest - GridSearchCV¶
In [153]:
%%time
# Creating pipeline
pipe = make_pipeline(StandardScaler(), RandomForestClassifier(random_state=1,class_weight={0:0.16,1:0.84}))
# Parameter grid to pass in GridSearchCV
param_grid = {
"randomforestclassifier__max_depth": np.arange(1,10),
"randomforestclassifier__n_estimators": [150,200,250],
"randomforestclassifier__min_samples_leaf": np.arange(5, 10),
"randomforestclassifier__max_features": np.arange(0.2, 0.7, 0.1)
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
# Calling GridSearchCV
grid_cv = GridSearchCV(estimator=pipe, param_grid=param_grid, scoring=scorer, cv=5)
# Fitting parameters in GridSeachCV
grid_cv.fit(X_train, y_train)
print(
"Best Parameters:{} \nScore: {}".format(grid_cv.best_params_, grid_cv.best_score_)
)
In [154]:
# Creating new pipeline with best parameters
rf_tuned1 = make_pipeline(
StandardScaler(),
RandomForestClassifier(
class_weight={0:0.16,1:0.84},random_state=1,
max_features=0.6000000000000001,
min_samples_leaf=6,
n_estimators=150,
max_depth=8
)
)
# Fit the model on training data
rf_tuned1.fit(X_train, y_train)
Out[154]:
In [155]:
# Calculating different metrics
get_metrics_score(rf_tuned1,X_train,X_test,y_train,y_test)
# Creating confusion matrix
make_confusion_matrix(rf_tuned1, y_test)
- Recall on training set is 96% and recall on test set is 90 %
- Precision and Accuracy scores have improved compared to Decision Tree tuned model using GridSearch
- Let us see if the difference b/w train and test recall scores can be improved further
- False Negative is low and that is really good for the bank
- True positives which are correct positive predictions is 14%, which is also very good for the model
Feature importance - Random Forest using GridSearch¶
In [156]:
feature_names = X_train.columns
importances = rf_tuned1[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Trans_Amt, Total_Revolving_bal are the 2 most important features impacting attrition as per Random Forest using GridSearch
- This model is not much over-fitting on training data and gives good scores (train,test scores are having difference though)
- False Negative is low and True positive is 14%, which is good
Bagging Classifier - GridSearchCV¶
In [157]:
%%time
# Creating pipeline
pipe = make_pipeline(StandardScaler(), BaggingClassifier(random_state=1))
# Parameter grid to pass in GridSearchCV
param_grid = {
'baggingclassifier__max_samples': np.arange(0.7,1.1,0.1),
'baggingclassifier__max_features': np.arange(0.7,1.1,0.1),
'baggingclassifier__n_estimators' : np.arange(10,60,10)
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
# Calling GridSearchCV
grid_cv = GridSearchCV(estimator=pipe, param_grid=param_grid, scoring=scorer, cv=5)
# Fitting parameters in GridSeachCV
grid_cv.fit(X_train, y_train)
print(
"Best Parameters:{} \nScore: {}".format(grid_cv.best_params_, grid_cv.best_score_)
)
In [158]:
# Creating new pipeline with best parameters
bc_tuned1 = make_pipeline(
StandardScaler(),
BaggingClassifier(
random_state=1,
max_features=0.9999999999999999,
max_samples=0.8999999999999999,
n_estimators=50
)
)
# Fit the model on training data
bc_tuned1.fit(X_train, y_train)
Out[158]:
In [159]:
# Calculating different metrics
get_metrics_score(bc_tuned1,X_train,X_test,y_train,y_test)
# Creating confusion matrix
make_confusion_matrix(bc_tuned1, y_test)
- Recall on training set is 99% and recall on test set is 83%
- The model shows over-fitting on training dataset
- False Negative increased and True positive decreased compared to the above 2 models
Bagging classifier has no feature importance attribute¶
XGBoost - GridSearchCV¶
In [160]:
%%time
# Creating pipeline
pipe = make_pipeline(StandardScaler(), XGBClassifier(random_state=1,eval_metric='logloss'))
# Parameter grid to pass in GridSearchCV
param_grid = {
'xgbclassifier__n_estimators': np.arange(75,175,25),
'xgbclassifier__subsample':np.arange(0.7,1.1,0.1),
'xgbclassifier__learning_rate':[0.01,0.1,0.2,0.05],
#'xgbclassifier__gamma':np.arange(0,6,1),
'xgbclassifier__colsample_bytree':np.arange(0.5,1.0,0.2),
'xgbclassifier__colsample_bylevel':np.arange(0.5,1.0,0.2),
'xgbclassifier__scale_pos_weight':np.arange(5,7,1)
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
# Calling GridSearchCV
grid_cv = GridSearchCV(estimator=pipe, param_grid=param_grid, scoring=scorer, cv=5)
# Fitting parameters in GridSeachCV
grid_cv.fit(X_train, y_train)
print(
"Best Parameters:{} \nScore: {}".format(grid_cv.best_params_, grid_cv.best_score_)
)
In [161]:
# Creating new pipeline with best parameters
xgb_tuned1 = make_pipeline(
StandardScaler(),
XGBClassifier(
random_state=1,
eval_metric='logloss',
n_estimators=100,
subsample=0.7,
learning_rate=0.05,
#gamma=5,
colsample_bytree=0.7,
colsample_bylevel=0.8999999999999999,
scale_pos_weight=6
)
)
# Fit the model on training data
xgb_tuned1.fit(X_train, y_train)
Out[161]:
In [162]:
# Calculating different metrics
get_metrics_score(xgb_tuned1,X_train,X_test,y_train,y_test)
# Creating confusion matrix
make_confusion_matrix(xgb_tuned1, y_test)
- Recall on training set is 99% and recall on test set is 94%
- Precision and Accuracy scores are good
- False Negative is low and that is really good for the bank
- True positives which are correct positive predictions is 15%, which is also very good for the model
Feature importance - XGBoost using GridSearch¶
In [163]:
feature_names = X_train.columns
importances = xgb_tuned1[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Revolving_bal,Total_Trans_Amt are the 2 most important features impacting attrition from XGBoost using GridSearch
- This model is not over-fitting on training data and gives good scores
- False Negative is very low, which is good
- True positive is 15% which is also good
Adaboost - GridSearchCV¶
In [164]:
%%time
# Creating pipeline
pipe = make_pipeline(StandardScaler(), AdaBoostClassifier(random_state=1))
# Parameter grid to pass in GridSearchCV
param_grid = {
"adaboostclassifier__n_estimators": np.arange(10, 110, 10),
"adaboostclassifier__learning_rate": [0.1, 0.01, 0.2, 0.05, 1],
"adaboostclassifier__base_estimator": [
DecisionTreeClassifier(max_depth=1, random_state=1),
DecisionTreeClassifier(max_depth=2, random_state=1),
DecisionTreeClassifier(max_depth=3, random_state=1),
]
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
# Calling GridSearchCV
grid_cv = GridSearchCV(estimator=pipe, param_grid=param_grid, scoring=scorer, cv=5)
# Fitting parameters in GridSeachCV
grid_cv.fit(X_train, y_train)
print(
"Best Parameters:{} \nScore: {}".format(grid_cv.best_params_, grid_cv.best_score_)
)
In [165]:
# Creating new pipeline with best parameters
abc_tuned1 = make_pipeline(
StandardScaler(),
AdaBoostClassifier(
random_state=1,
n_estimators=70,
learning_rate=1,
base_estimator=DecisionTreeClassifier(max_depth=2, random_state=1)
)
)
# Fit the model on training data
abc_tuned1.fit(X_train, y_train)
Out[165]:
In [166]:
# Calculating different metrics
get_metrics_score(abc_tuned1,X_train,X_test,y_train,y_test)
# Creating confusion matrix
make_confusion_matrix(abc_tuned1, y_test)
- Recall on training set is 91% and recall on test set is 86 %
- Precision and Accuracy scores are good, will see if the train Vs test recall difference is improved with other models
- False Negative slightly went up compared to XGBoost with GridSearch
- True positive slightly went down compared to XGBoost with GridSearch
Feature importance - Adaboost using GridSearch¶
In [167]:
feature_names = X_train.columns
importances = abc_tuned1[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Trans_Amt, Change in Transaction Amount (Q4 over Q1) are the 2 most important features impacting attrition from Adaboost using GridSearch
- This model is not over-fitting on training data and gives good scores
- False Negative slightly went up compared to XGBoost with GridSearch
- True positive slightly went down compared to XGBoost with GridSearch
Gradientboost - GridSearchCV¶
In [168]:
%%time
# Creating pipeline
pipe = make_pipeline(StandardScaler(), GradientBoostingClassifier(init=AdaBoostClassifier(random_state=1),random_state=1))
# Parameter grid to pass in GridSearchCV
param_grid = {
'gradientboostingclassifier__n_estimators': np.arange(50,200,25),
'gradientboostingclassifier__subsample':np.arange(0.7,1.1,0.1),
'gradientboostingclassifier__max_features':np.arange(0.7,1.1,0.1)
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
# Calling GridSearchCV
grid_cv = GridSearchCV(estimator=pipe, param_grid=param_grid, scoring=scorer, cv=5)
# Fitting parameters in GridSeachCV
grid_cv.fit(X_train, y_train)
print(
"Best Parameters:{} \nScore: {}".format(grid_cv.best_params_, grid_cv.best_score_)
)
In [169]:
# Creating new pipeline with best parameters
gbm_tuned1 = make_pipeline(
StandardScaler(),
GradientBoostingClassifier(
random_state=1,
n_estimators=175,
subsample=0.7999999999999999,
max_features=0.8999999999999999,
init=AdaBoostClassifier(random_state=1)
)
)
# Fit the model on training data
gbm_tuned1.fit(X_train, y_train)
Out[169]:
In [170]:
# Calculating different metrics
get_metrics_score(gbm_tuned1,X_train,X_test,y_train,y_test)
# Creating confusion matrix
make_confusion_matrix(gbm_tuned1, y_test)
- Recall on training set is 88% and recall on test set is 84 %
- Precision and Accuracy scores are good too
- False Negative slightly went up compared to XGBoost with GridSearch
- True positive slightly went down compared to XGBoost with GridSearch
Feature importance - GradientBoost using GridSearch¶
In [171]:
feature_names = X_train.columns
importances = gbm_tuned1[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Trans_Amt, Total_Revolving_bal are the 2 most important features impacting attrition from Gradient Boost using GridSearch
- This model is not over-fitting on training data and gives good scores
- False Negative slightly went up compared to XGBoost with GridSearch
- True positive slightly went down compared to XGBoost with GridSearch
Hyperparameter Tuning Using RandomSearch¶
- Using pipelines with StandardScaler and each of the above model to tune the model using RandomizedSearchCV
- Using make_pipeline() function to create pipeline
Decision Tree - RandomizedSearchCV¶
In [172]:
%%time
#Creating pipeline
pipe=make_pipeline(StandardScaler(),DecisionTreeClassifier(criterion='gini',class_weight={0:0.16,1:0.84},random_state=1))
#Parameter distribution to pass in RandomSearchCV
param_dist={
'decisiontreeclassifier__max_depth': np.arange(1,10),
'decisiontreeclassifier__min_samples_leaf': np.arange(1,10,2),
'decisiontreeclassifier__max_leaf_nodes' : np.arange(1,10,2),
'decisiontreeclassifier__min_impurity_decrease': [0.0001,0.001,0.01],
'decisiontreeclassifier__max_features': np.arange(0.2, 0.7, 0.2)
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
#Calling RandomizedSearchCV
randomized_cv = RandomizedSearchCV(estimator=pipe, param_distributions=param_dist, n_iter=50, scoring=scorer, cv=5, random_state=1)
#Fitting parameters in RandomizedSearchCV
randomized_cv.fit(X_train,y_train)
print("Best parameters are {} with CV score={}:" .format(randomized_cv.best_params_,randomized_cv.best_score_))
In [173]:
# Creating new pipeline with best parameters
dtree_tuned2 = make_pipeline(
StandardScaler(),
DecisionTreeClassifier(
criterion='gini',class_weight={0:0.16,1:0.84},random_state=1,
max_depth=9,
min_samples_leaf=3,
min_impurity_decrease=0.0001,
max_leaf_nodes=9,
max_features=0.6000000000000001
)
)
# Fit the model on training data
dtree_tuned2.fit(X_train, y_train)
Out[173]:
In [174]:
# Calculating different metrics
get_metrics_score(dtree_tuned2,X_train,X_test,y_train,y_test)
# Creating confusion matrix
make_confusion_matrix(dtree_tuned2, y_test)
- Recall on training set is 86% and recall on test set is 84%
- Which is good, since it is not showing overfit and gives good results
- But precision scores can be improved
- Looking at the confusion matrix, False Negative is low, which is good, as explained in "how to interpret confusion matrix"
- True positives which are correct positive predictions is 13%, which is also good for the model
Feature importance - Decision Tree using Random Search¶
In [175]:
feature_names = X_train.columns
importances = dtree_tuned2[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Trans_Amt,Change in Transaction Count (Q4 over Q1),Total_Revolving_Bal are the most important features impacting attrition as per Decision Tree using Random Search
- This model is not over-fitting on training data and gives good scores
- False Negative is low and True positive is 13% which is good
Random Forest - RandomizedSearchCV¶
In [176]:
%%time
#Creating pipeline
pipe=make_pipeline(StandardScaler(),RandomForestClassifier(random_state=1,class_weight={0:0.16,1:0.84}))
#Parameter distribution to pass in RandomSearchCV
param_dist= {
"randomforestclassifier__max_depth": np.arange(1,10),
"randomforestclassifier__n_estimators": [150,200,250],
"randomforestclassifier__min_samples_leaf": np.arange(5, 10),
"randomforestclassifier__max_features": np.arange(0.2, 0.7, 0.1)
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
#Calling RandomizedSearchCV
randomized_cv = RandomizedSearchCV(estimator=pipe, param_distributions=param_dist, n_iter=50, scoring=scorer, cv=5, random_state=1)
#Fitting parameters in RandomizedSearchCV
randomized_cv.fit(X_train,y_train)
print("Best parameters are {} with CV score={}:" .format(randomized_cv.best_params_,randomized_cv.best_score_))
In [177]:
# Creating new pipeline with best parameters
rf_tuned2 = make_pipeline(
StandardScaler(),
RandomForestClassifier(
class_weight={0:0.16,1:0.84},
random_state=1,
max_features=0.6000000000000001,
min_samples_leaf=9,
n_estimators=150,
max_depth=9
)
)
# Fit the model on training data
rf_tuned2.fit(X_train, y_train)
Out[177]:
In [178]:
# Calculating different metrics
get_metrics_score(rf_tuned2,X_train,X_test,y_train,y_test)
# Creating confusion matrix
make_confusion_matrix(rf_tuned2, y_test)
- Recall on training set is 96% and recall on test set is 90%
- Precision and Accuracy scores have improved compared to Decision Tree tuned model using RandomSearch
- Looking at the confusion matrix, False Negative is low, which is good
- True positives which are correct positive predictions is 14%, which is also very good for the model
Feature importance - Random Forest using Random Search¶
In [179]:
feature_names = X_train.columns
importances = rf_tuned2[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Trans_Amt, Total_Revolving_bal are the 2 most important features impacting attrition from Random Forest using Random Search
- This model is not over-fitting on training data and gives good scores
- False Negative is low and True positive is 14% which is good
Bagging Classifier - RandomizedSearchCV¶
In [180]:
%%time
#Creating pipeline
pipe=make_pipeline(StandardScaler(),BaggingClassifier(random_state=1))
#Parameter distribution to pass in RandomSearchCV
param_dist= {
'baggingclassifier__max_samples': np.arange(0.7,1.1,0.1),
'baggingclassifier__max_features': np.arange(0.7,1.1,0.1),
'baggingclassifier__n_estimators' : np.arange(10,60,10)
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
#Calling RandomizedSearchCV
randomized_cv = RandomizedSearchCV(estimator=pipe, param_distributions=param_dist, n_iter=50, scoring=scorer, cv=5, random_state=1)
#Fitting parameters in RandomizedSearchCV
randomized_cv.fit(X_train,y_train)
print("Best parameters are {} with CV score={}:" .format(randomized_cv.best_params_,randomized_cv.best_score_))
In [181]:
# Creating new pipeline with best parameters
bc_tuned2 = make_pipeline(
StandardScaler(),
BaggingClassifier(
random_state=1,
max_features=0.9999999999999999,
max_samples=0.7999999999999999,
n_estimators=40
)
)
# Fit the model on training data
bc_tuned2.fit(X_train, y_train)
Out[181]:
In [182]:
# Calculating different metrics
get_metrics_score(bc_tuned2,X_train,X_test,y_train,y_test)
# Creating confusion matrix
make_confusion_matrix(bc_tuned2, y_test)
- Recall on training set is 98% and recall on test set is 80%
- The model shows over-fitting on training dataset
- Looking at the confusion matrix, False Negative is high compared to all other models so far
- True positives which are correct positive predictions is 13%, which is low compared to other models so far
Bagging Classifier has no feature importance attribute¶
XGBoost - RandomizedSearchCV¶
In [183]:
%%time
#Creating pipeline
pipe=make_pipeline(StandardScaler(),XGBClassifier(random_state=1,eval_metric='logloss'))
#Parameter distribution to pass in RandomSearchCV
param_dist= {
'xgbclassifier__n_estimators': np.arange(75,175,25),
'xgbclassifier__subsample':np.arange(0.7,1.1,0.1),
'xgbclassifier__learning_rate':[0.01,0.1,0.2,0.05],
#'xgbclassifier__gamma':np.arange(0,6,1),
'xgbclassifier__colsample_bytree':np.arange(0.5,1.0,0.2),
'xgbclassifier__colsample_bylevel':np.arange(0.5,1.0,0.2),
'xgbclassifier__scale_pos_weight':np.arange(5,7,1)
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
#Calling RandomizedSearchCV
randomized_cv = RandomizedSearchCV(estimator=pipe, param_distributions=param_dist, n_iter=50, scoring=scorer, cv=5, random_state=1)
#Fitting parameters in RandomizedSearchCV
randomized_cv.fit(X_train,y_train)
print("Best parameters are {} with CV score={}:" .format(randomized_cv.best_params_,randomized_cv.best_score_))
In [184]:
# Creating new pipeline with best parameters
xgb_tuned2 = make_pipeline(
StandardScaler(),
XGBClassifier(
random_state=1,
eval_metric='logloss',
n_estimators=75,
subsample=0.7,
learning_rate=0.1,
#gamma=5,
colsample_bytree=0.5,
colsample_bylevel=0.8999999999999999,
scale_pos_weight=6
)
)
# Fit the model on training data
xgb_tuned2.fit(X_train, y_train)
Out[184]:
In [185]:
# Calculating different metrics
get_metrics_score(xgb_tuned2,X_train,X_test,y_train,y_test)
# Creating confusion matrix
make_confusion_matrix(xgb_tuned2, y_test)
- Recall on training set is 99% and recall on test set is 94 %
- Precision and Accuracy scores are good
- Looking at the confusion matrix, False Negative is very low, which is good, as explained in "how to interpret confusion matrix"
- True positives which are correct positive predictions is 15%, which is also very good for the model
Feature importance - XGBoost using Random Search¶
In [186]:
feature_names = X_train.columns
importances = xgb_tuned2[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Trans_Amt, Total_Revolving_bal are the 2 most important features impacting attrition from XGBoost using Random Search
- This model is not over-fitting on training data and gives good scores
- False Negative is very low and true positive is higher, which is good
Adaboost - RandomizedSearchCV¶
In [187]:
%%time
#Creating pipeline
pipe=make_pipeline(StandardScaler(),AdaBoostClassifier(random_state=1))
#Parameter distribution to pass in RandomSearchCV
param_dist= {
"adaboostclassifier__n_estimators": np.arange(10, 110, 10),
"adaboostclassifier__learning_rate": [0.1, 0.01, 0.2, 0.05, 1],
"adaboostclassifier__base_estimator": [
DecisionTreeClassifier(max_depth=1, random_state=1),
DecisionTreeClassifier(max_depth=2, random_state=1),
DecisionTreeClassifier(max_depth=3, random_state=1),
]
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
#Calling RandomizedSearchCV
randomized_cv = RandomizedSearchCV(estimator=pipe, param_distributions=param_dist, n_iter=50, scoring=scorer, cv=5, random_state=1)
#Fitting parameters in RandomizedSearchCV
randomized_cv.fit(X_train,y_train)
print("Best parameters are {} with CV score={}:" .format(randomized_cv.best_params_,randomized_cv.best_score_))
In [188]:
# Creating new pipeline with best parameters
abc_tuned2 = make_pipeline(
StandardScaler(),
AdaBoostClassifier(
random_state=1,
n_estimators=90,
learning_rate=0.2,
base_estimator=DecisionTreeClassifier(max_depth=3, random_state=1)
)
)
# Fit the model on training data
abc_tuned2.fit(X_train, y_train)
Out[188]:
In [189]:
# Calculating different metrics
get_metrics_score(abc_tuned2,X_train,X_test,y_train,y_test)
# Creating confusion matrix
make_confusion_matrix(abc_tuned2, y_test)
- Recall on training set is 92% and recall on test set is 85 %
- Precision and Accuracy scores are good too
- Looking at the confusion matrix, False Negative is low, which is good, bit more compared to xgboost
- True positives which are correct positive predictions is 13%, slightly lower than xgboost
Feature importance - Adaboost using Random Search¶
In [190]:
feature_names = X_train.columns
importances = abc_tuned2[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Trans_Amt, Change in Transaction Amount (Q4 over Q1) are the 2 most important features impacting attrition from Adaboost using Random Search
- This model is not over-fitting on training data and gives good scores
- Looking at the confusion matrix, False Negative is low, which is good, bit more compared to xgboost
- True positives which are correct positive predictions is 13%, slightly lower than xgboost
GradientBoost - RandomizedSearchCV¶
In [191]:
%%time
#Creating pipeline
pipe=make_pipeline(StandardScaler(),GradientBoostingClassifier(init=AdaBoostClassifier(random_state=1),random_state=1))
#Parameter distribution to pass in RandomSearchCV
param_dist= {
'gradientboostingclassifier__n_estimators': np.arange(50,200,25),
'gradientboostingclassifier__subsample':np.arange(0.7,1.1,0.1),
'gradientboostingclassifier__max_features':np.arange(0.7,1.1,0.1)
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.recall_score)
#Calling RandomizedSearchCV
randomized_cv = RandomizedSearchCV(estimator=pipe, param_distributions=param_dist, n_iter=50, scoring=scorer, cv=5, random_state=1)
#Fitting parameters in RandomizedSearchCV
randomized_cv.fit(X_train,y_train)
print("Best parameters are {} with CV score={}:" .format(randomized_cv.best_params_,randomized_cv.best_score_))
In [192]:
# Creating new pipeline with best parameters
gbm_tuned2 = make_pipeline(
StandardScaler(),
GradientBoostingClassifier(
random_state=1,
n_estimators=175,
subsample=0.7999999999999999,
max_features=0.7999999999999999,
init=AdaBoostClassifier(random_state=1)
)
)
# Fit the model on training data
gbm_tuned2.fit(X_train, y_train)
Out[192]:
In [193]:
# Calculating different metrics
get_metrics_score(gbm_tuned2,X_train,X_test,y_train,y_test)
# Creating confusion matrix
make_confusion_matrix(gbm_tuned2, y_test)
- Recall on training set is 89% and recall on test set is 84 %
- Precision and Accuracy scores are good
- Looking at the confusion matrix, False Negative is low, which is good, as explained in "how to interpret confusion matrix"
- True positives which are correct positive predictions is 13.5%, which is also good for the model
Feature importance - GradientBoost using Random Search¶
In [194]:
feature_names = X_train.columns
importances = gbm_tuned2[1].feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12, 12))
plt.title("Feature Importances")
plt.barh(range(len(indices)), importances[indices], color="violet", align="center")
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel("Relative Importance")
plt.show()
- Total_Trans_Amt, Total_Revolving_bal are the 2 most important features impacting attrition from Gradient Boost using Random Search
- This model is not over-fitting on training data and gives good scores
- False negative and true positive also good from confusion matrix
Comparing models¶
In [195]:
# defining list of model - logistic regression
models = [lr]
# defining empty lists to add train and test results
acc_train = []
acc_test = []
recall_train = []
recall_test = []
precision_train = []
precision_test = []
# looping through all the models to get the metrics score - Accuracy, Recall and Precision
for model in models:
j = get_metrics_score(model,X_train,X_test,y_train,y_test,False)
acc_train.append(j[0])
acc_test.append(j[1])
recall_train.append(j[2])
recall_test.append(j[3])
precision_train.append(j[4])
precision_test.append(j[5])
In [196]:
# defining list of models - logistic regression over-sampling,regularization
models = [log_reg_over, lr_estimator]
# looping through all the models to get the metrics score - Accuracy, Recall and Precision
for model in models:
j = get_metrics_score(model,X_train_over,X_test,y_train_over,y_test,False)
acc_train.append(j[0])
acc_test.append(j[1])
recall_train.append(j[2])
recall_test.append(j[3])
precision_train.append(j[4])
precision_test.append(j[5])
In [197]:
# defining list of model - logistic regression undersampling
models = [log_reg_under]
# looping through all the models to get the metrics score - Accuracy, Recall and Precision
for model in models:
j = get_metrics_score(model,X_train_un,X_test,y_train_un,y_test,False)
acc_train.append(j[0])
acc_test.append(j[1])
recall_train.append(j[2])
recall_test.append(j[3])
precision_train.append(j[4])
precision_test.append(j[5])
In [198]:
# defining list of model - Bagging/Boosting without hyper parameter tuning
models = [dtree,rf,bc,xgb,abc,gbm]
# looping through all the models to get the metrics score - Accuracy, Recall and Precision
for model in models:
j = get_metrics_score(model,X_train_un,X_test,y_train_un,y_test,False)
acc_train.append(j[0])
acc_test.append(j[1])
recall_train.append(j[2])
recall_test.append(j[3])
precision_train.append(j[4])
precision_test.append(j[5])
In [199]:
# defining list of model - Bagging/Boosting with hyper parameter tuning using GridSearch
models = [dtree_tuned1,rf_tuned1,bc_tuned1,xgb_tuned1,abc_tuned1,gbm_tuned1]
# looping through all the models to get the metrics score - Accuracy, Recall and Precision
for model in models:
j = get_metrics_score(model,X_train_un,X_test,y_train_un,y_test,False)
acc_train.append(j[0])
acc_test.append(j[1])
recall_train.append(j[2])
recall_test.append(j[3])
precision_train.append(j[4])
precision_test.append(j[5])
In [200]:
# defining list of model - Bagging/Boosting with hyper parameter tuning using RandomSearch
models = [dtree_tuned2,rf_tuned2,bc_tuned2,xgb_tuned2,abc_tuned2,gbm_tuned2]
# looping through all the models to get the metrics score - Accuracy, Recall and Precision
for model in models:
j = get_metrics_score(model,X_train_un,X_test,y_train_un,y_test,False)
acc_train.append(j[0])
acc_test.append(j[1])
recall_train.append(j[2])
recall_test.append(j[3])
precision_train.append(j[4])
precision_test.append(j[5])
In [201]:
comparison_frame = pd.DataFrame(
{
'Model':['Logistic Regression',
'Logistic Regression - Oversampled data',
'Logistic Regression-Regularized-OversampledData',
'Logistic Regression - Undersampled data',
'Decision Tree Classifier',
'RandomForest Classifier',
'Bagging Classifier',
'XGBoost Classifier',
'Adaboost Classifier',
'GradientBoost Classifier',
'Decision Tree - GridSearch',
'RandomForest - GridSearch',
'Bagging - GridSearch',
'XGBoost - GridSearch',
'Adaboost - GridSearch',
'GradientBoost - GridSearch',
'Decision Tree - RandomSearch',
'RandomForest - RandomSearch',
'Bagging - RandomSearch',
'XGBoost - RandomSearch',
'Adaboost - RandomSearch',
'GradientBoost - RandomSearch'
],
'Train_Accuracy': acc_train,'Test_Accuracy': acc_test,
'Train_Recall':recall_train,'Test_Recall':recall_test,
'Train_Precision':precision_train,'Test_Precision':precision_test
}
)
#Sorting models in decreasing order of test recall
comparison_frame.sort_values(by="Test_Recall", ascending=False)
Out[201]:
- Based on test recall, the best model is XGBoost - RandomSearch, followed by XGBoost - GridSearch
- Decision Tree - GridSearch shows 3rd best test recall, but the precision score is very low
- RandomForest - RandomSearch and RandomForest - GridSearch show good test recall and precision scores and hence is better than decision tree - gridsearch
- Models that over-fit are XGBoost Classifier,Decision Tree Classifier,RandomForest Classifier and Bagging Classifier (train and test scores are not close)
- When we consider Logistic Regression models alone, undersampled model gives better preformance than other LR models. But compared to other models, LR models perform poor for this classification problem
- When we consider base models alone, XGBoost performs better than other models
- Even though the focus for this classification problem is recall score, we should be choosing a model that gives better precision too, since that results in more resources. In that aspect, XGBoost - RandomSearch gives better precision score too
- We can also see that hyper-parameter tuning definitely improves the performance of models, when carefully done, without over-fitting
- For a given set of parameters, random search runs faster than grid search
Top 5 features impacting attrition based on xgboost random search model are¶
- Total_Trans_Amt (Total Transaction Amount last 12 months)
- Total_Revolving_Bal (Total Revolving Balance or unpaid amount carried over to next billing cycle on the Credit Card)
- Total_Ct_Chng_Q4_Q1 (Change in Transaction Count Q4 over Q1)
- Total_Amt_Chng_Q4_Q1 (Change in Transaction Amount Q4 over Q1)
- Total_Relationship_Count (Total no. of products held by the customer)
Time taken by grid and randomized search¶
In [203]:
time_df = pd.DataFrame(
{
'Model':[
'Decision Tree',
'Random Forest',
'Bagging Classifier',
'XGBoost',
'Adaboost',
'GradientBoost'
],
'Grid Search': ['2min 49s','1h 7 min 19s','4min 15s','44min 32s','7min 3s','10min 41s'],
'Randomized Search': ['4.27s','5min 1s','1min 31s','1min 18s','2min 28s','3min 33s']
}
)
time_df
Out[203]:
- We can see that, Random Search always takes less time compared to Grid Search
- This is because, Grid Search tries out all possible combinations of hyper parameters given and finds the best from the given set, whereas Random Search tries random number of samples (equal to n_iter) and chooses the best one
- And from the model analysis we have seen that Random Search can also provide better result by doing random sample search
- Random Forest and XGBoost GridSearch take longer compared to other models
NOTE on hyper parameter tuning¶
- For each model, same parameter grid/distribution is used between GridSearch and RandomSearch to get a true compare between the two. For example, XGBoost GridSearch and XGBoost RandomSearch uses same set of hyper parameters
- Ideally for RandomSearch we should be giving a parameter distribution of continuous values
- But in this case, since we need to perform a true compare between GridSearch and RandomSearch, we should be using same set of parameters for both
- Using a wide range of parameter values for GridSearch was taking really long especially for XGBoost, Random Forest etc. and sometimes takes forever
- So due to limitation in computation efficiency, I had to use discrete values for RandomSearch in some cases
Business recommendations and insights¶
- This model can be used to predict if a customer is going to leave credit card services or not, provided the feature set is going to be the same. Even though the model gives more focus on false negative predictions (model predicts customer will NOT leave credit card services, but customer actually leaves credit card services), there is some insight towards false positive predictions (model predicts customer will leave credit card services, but customer actually does NOT leave credit card services) too, which in turn causes the bank extra resources.
- Based on the analysis done by using different models, these are some of the key features bank can focus on while deciding if a customer is going to leave credit card services or not
- Total_Trans_Amt (Total Transaction Amount last 12 months)
- No. of Contacts in the last 12 months
- No. of months inactive in the last 12 months
- Total_Revolving_Bal (Total Revolving Balance or unpaid amount carried over to next billing cycle on the Credit Card)
- Total_Ct_Chng_Q4_Q1 (Change in Transaction Count Q4 over Q1)
- Total_Amt_Chng_Q4_Q1 (Change in Transaction Amount Q4 over Q1)
- Total_Relationship_Count (Total no. of products held by the customer)
- Total Transaction Count (Last 12 months)
- Avg_Utilization_Ratio (Average Card Utilization Ratio)
- Bank can focus more on customers having more total revolving balance, since these customers tend to keep using credit card services
- Bank can focus more on customers having more total transaction amount in last 12 months, since these customers tend to keep using credit card services
- Bank can focus more on customers having more total transaction count in last 12 months, since these customers tend to keep using credit card services
- Bank can focus more on customers having more card utilization, since these customers tend to keep using credit card services
- Bank can focus more on customers having more products offered by the bank, since these customers tend to keep using credit card services
- Bank should keep an eye on customers not using the card for months (number of inactive months), since when the card is inactive for longer period, those customers tend to leave credit card services
- Bank should keep an eye on customers who need to be contacted more (number of contacts b/w bank and customer), since those customers seem to be leaving credit card services more
- Bank can see why Platinum card shows bit more attrition rate than other tyeps of cards
- Bank can see why attrition rate is slightly more for females,even though the difference is not significant
- Bank can increase credit limit for the above scenarios where there is more chances of a customer leaving credit card services, which in turn increases open to buy credit line, and increases the chance of customers sticking to credit card services
- Since Platinum card shows bit more attrition rate than other tyeps of cards, additional benefits can be offered for Platinum card like cash back etc.
- Female customers can be given shopping offers, since the attrition rate seems to be slightly higher for them
- Similarly attrition rate seems to be slightly more for education level = doctorate, some kind of offer/discount can be given to them to make them keep using credit card services